Finite size e ects in on - line learning of multi - layerneural networks

We complement the recent progress in thermodynamic limit analyses of mean on-line gradient descent learning dynamics in multi-layer networks by calculating the uctuations possessed by nite dimensional systems. Fluctuations from the mean dynamics are largest at the onset of specialisation as student hidden unit weight vectors begin to imitate speciic teacher vectors, and increase with the degree of symmetry of the initial conditions. Including a term to stimulate asymmetry in the learning process typically signiicantly decreases nite size eeects and training time. Recent advances in the theory of on-line learning have yielded insights into the training dynamics of multi-layer neural networks. In on-line learning, the weights parametrizing the student network are updated according to the error on a single example from a stream of examples, f ; ()g, generated by a teacher network ())1]. The analysis of the resulting weight dynamics has previously been treated by assuming an innnite input dimension (thermodynamic limit) such that a mean dynamics analysis is exactt2]. We present a more realistic treatment by calculating corrections to the mean dynamics induced by nite dimensional inputss3]. We assume that the teacher network the student attempts to learn is a soft committee machine1] of N inputs, and M hidden units, this being a one hidden layer network with weights connecting each hidden to output unit set to +1, and with each hidden unit n connected to all input units by B n (n = 1::M). Explicitly, for the N dimensional training input vector , the output of the teacher is given by, = M X n=1 g(B n); (1) where g(x) is the activation function of the hidden units, and we take g(x) = erf(x= p 2). The teacher generates a stream of training examples (;), with input components drawn from a normal distribution of zero mean, unit variance.